{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n================================================================\nPermutation Importance vs Random Forest Feature Importance (MDI)\n================================================================\n\nIn this example, we will compare the impurity-based feature importance of\n:class:`~sklearn.ensemble.RandomForestClassifier` with the\npermutation importance on the titanic dataset using\n:func:`~sklearn.inspection.permutation_importance`. We will show that the\nimpurity-based feature importance can inflate the importance of numerical\nfeatures.\n\nFurthermore, the impurity-based feature importance of random forests suffers\nfrom being computed on statistics derived from the training dataset: the\nimportances can be high even for features that are not predictive of the target\nvariable, as long as the model has the capacity to use them to overfit.\n\nThis example shows how to use Permutation Importances as an alternative that\ncan mitigate those limitations.\n\n.. topic:: References:\n\n   [1] L. Breiman, \"Random Forests\", Machine Learning, 45(1), 5-32,\n       2001. https://doi.org/10.1023/A:1010933404324\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(__doc__)\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom sklearn.datasets import fetch_openml\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.inspection import permutation_importance\nfrom sklearn.compose import ColumnTransformer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.preprocessing import OneHotEncoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Data Loading and Feature Engineering\n------------------------------------\nLet's use pandas to load a copy of the titanic dataset. The following shows\nhow to apply separate preprocessing on numerical and categorical features.\n\nWe further include two random variables that are not correlated in any way\nwith the target variable (``survived``):\n\n- ``random_num`` is a high cardinality numerical variable (as many unique\n  values as records).\n- ``random_cat`` is a low cardinality categorical variable (3 possible\n  values).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "X, y = fetch_openml(\"titanic\", version=1, as_frame=True, return_X_y=True)\nrng = np.random.RandomState(seed=42)\nX['random_cat'] = rng.randint(3, size=X.shape[0])\nX['random_num'] = rng.randn(X.shape[0])\n\ncategorical_columns = ['pclass', 'sex', 'embarked', 'random_cat']\nnumerical_columns = ['age', 'sibsp', 'parch', 'fare', 'random_num']\n\nX = X[categorical_columns + numerical_columns]\n\nX_train, X_test, y_train, y_test = train_test_split(\n    X, y, stratify=y, random_state=42)\n\ncategorical_pipe = Pipeline([\n    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n    ('onehot', OneHotEncoder(handle_unknown='ignore'))\n])\nnumerical_pipe = Pipeline([\n    ('imputer', SimpleImputer(strategy='mean'))\n])\n\npreprocessing = ColumnTransformer(\n    [('cat', categorical_pipe, categorical_columns),\n     ('num', numerical_pipe, numerical_columns)])\n\nrf = Pipeline([\n    ('preprocess', preprocessing),\n    ('classifier', RandomForestClassifier(random_state=42))\n])\nrf.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Accuracy of the Model\n---------------------\nPrior to inspecting the feature importances, it is important to check that\nthe model predictive performance is high enough. Indeed there would be little\ninterest of inspecting the important features of a non-predictive model.\n\nHere one can observe that the train accuracy is very high (the forest model\nhas enough capacity to completely memorize the training set) but it can still\ngeneralize well enough to the test set thanks to the built-in bagging of\nrandom forests.\n\nIt might be possible to trade some accuracy on the training set for a\nslightly better accuracy on the test set by limiting the capacity of the\ntrees (for instance by setting ``min_samples_leaf=5`` or\n``min_samples_leaf=10``) so as to limit overfitting while not introducing too\nmuch underfitting.\n\nHowever let's keep our high capacity random forest model for now so as to\nillustrate some pitfalls with feature importance on variables with many\nunique values.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "print(\"RF train accuracy: %0.3f\" % rf.score(X_train, y_train))\nprint(\"RF test accuracy: %0.3f\" % rf.score(X_test, y_test))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Tree's Feature Importance from Mean Decrease in Impurity (MDI)\n--------------------------------------------------------------\nThe impurity-based feature importance ranks the numerical features to be the\nmost important features. As a result, the non-predictive ``random_num``\nvariable is ranked the most important!\n\nThis problem stems from two limitations of impurity-based feature\nimportances:\n\n- impurity-based importances are biased towards high cardinality features;\n- impurity-based importances are computed on training set statistics and\n  therefore do not reflect the ability of feature to be useful to make\n  predictions that generalize to the test set (when the model has enough\n  capacity).\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "ohe = (rf.named_steps['preprocess']\n         .named_transformers_['cat']\n         .named_steps['onehot'])\nfeature_names = ohe.get_feature_names(input_features=categorical_columns)\nfeature_names = np.r_[feature_names, numerical_columns]\n\ntree_feature_importances = (\n    rf.named_steps['classifier'].feature_importances_)\nsorted_idx = tree_feature_importances.argsort()\n\ny_ticks = np.arange(0, len(feature_names))\nfig, ax = plt.subplots()\nax.barh(y_ticks, tree_feature_importances[sorted_idx])\nax.set_yticklabels(feature_names[sorted_idx])\nax.set_yticks(y_ticks)\nax.set_title(\"Random Forest Feature Importances (MDI)\")\nfig.tight_layout()\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As an alternative, the permutation importances of ``rf`` are computed on a\nheld out test set. This shows that the low cardinality categorical feature,\n``sex`` is the most important feature.\n\nAlso note that both random features have very low importances (close to 0) as\nexpected.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = permutation_importance(rf, X_test, y_test, n_repeats=10,\n                                random_state=42, n_jobs=2)\nsorted_idx = result.importances_mean.argsort()\n\nfig, ax = plt.subplots()\nax.boxplot(result.importances[sorted_idx].T,\n           vert=False, labels=X_test.columns[sorted_idx])\nax.set_title(\"Permutation Importances (test set)\")\nfig.tight_layout()\nplt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "It is also possible to compute the permutation importances on the training\nset. This reveals that ``random_num`` gets a significantly higher importance\nranking than when computed on the test set. The difference between those two\nplots is a confirmation that the RF model has enough capacity to use that\nrandom numerical feature to overfit. You can further confirm this by\nre-running this example with constrained RF with min_samples_leaf=10.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "result = permutation_importance(rf, X_train, y_train, n_repeats=10,\n                                random_state=42, n_jobs=2)\nsorted_idx = result.importances_mean.argsort()\n\nfig, ax = plt.subplots()\nax.boxplot(result.importances[sorted_idx].T,\n           vert=False, labels=X_train.columns[sorted_idx])\nax.set_title(\"Permutation Importances (train set)\")\nfig.tight_layout()\nplt.show()"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}